{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating a Multi-Object Tracking Model on MOT Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides a framework for evaluating [FairMOT](https://github.com/ifzhang/FairMOT) on the [MOT Challenge dataset](https://motchallenge.net/).\n",
    "\n",
    "The MOT Challenge datasets are some of the most common benchmarking datasets for measuring multi-object tracking performance on pedestrian data. They provide distinct datasets every few years; their current offerings include MOT15, MOT16/17, and MOT 19/20. These datasets contain various annotated video sequences, each with different tracking difficulties. Additionally, MOT Challenge provides detections for tracking algorithms without detection components.\n",
    "\n",
    "The goal of this notebook is to re-produce published results on the MOT challenge using the state-of-the-art FairMOT approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure edits to libraries are loaded and plotting is shown in the notebook.\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TorchVision: 0.4.0a0+6b959ee\n",
      "Torch is using GPU: Tesla K80\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import os.path as osp\n",
    "import sys\n",
    "import time\n",
    "\n",
    "from urllib.parse import urljoin\n",
    "import torch\n",
    "import torchvision\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "from utils_cv.common.data import data_path, download, unzip_url\n",
    "from utils_cv.common.gpu import which_processor, is_windows\n",
    "from utils_cv.tracking.data import Urls\n",
    "from utils_cv.tracking.dataset import TrackingDataset\n",
    "from utils_cv.tracking.model import TrackingLearner\n",
    "\n",
    "# Change matplotlib backend so that plots are shown for windows\n",
    "if is_windows():\n",
    "    plt.switch_backend(\"TkAgg\")\n",
    "\n",
    "print(f\"TorchVision: {torchvision.__version__}\")\n",
    "which_processor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above torchvision command displays your machine's GPUs (if it has any) and the compute that `torch/torchvision` is using."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will set some model runtime parameters. Here we will specify the default FairMOT model (dla34) and will evaluate against the MOT17 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using torch device: cuda\n"
     ]
    }
   ],
   "source": [
    "CONF_THRES = 0.4\n",
    "TRACK_BUFFER = 30\n",
    "\n",
    "# Downloaded MOT Challendage data path\n",
    "MOT_ROOT_PATH = \"../../data/\"\n",
    "RESULT_ROOT = \"./results\"\n",
    "EXP_NAME = \"MOT_val_all_dla34\"\n",
    "\n",
    "BASELINE_MODEL = \"./models/all_dla34.pth\"\n",
    "MOTCHALLENGE_BASE_URL = \"https://motchallenge.net/data/\"\n",
    "\n",
    "# train on the GPU or on the CPU, if a GPU is not available\n",
    "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
    "print(f\"Using torch device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model and Dataset Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will download the [MOT17](https://motchallenge.net/data/MOT17.zip) dataset and save it to `MOT_SAVED_PATH`. Note: MOT17 is around 5GB and it may take some time to download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data saved to ../../data/MOT17/train\n",
      "Test data saved to ../../data/MOT17/test\n"
     ]
    }
   ],
   "source": [
    "mot_path = urljoin(MOTCHALLENGE_BASE_URL, \"MOT17.zip\")\n",
    "mot_train_path = osp.join(MOT_ROOT_PATH, \"MOT17\", \"train\")\n",
    "mot_test_path = osp.join(MOT_ROOT_PATH, \"MOT17\", \"test\")\n",
    "# seqs_str:  various video sequences subfolder names under MOT challenge data\n",
    "train_seqs = [\n",
    "    \"MOT17-02-SDP\",\n",
    "    \"MOT17-04-SDP\",\n",
    "    \"MOT17-05-SDP\",\n",
    "    \"MOT17-09-SDP\",\n",
    "    \"MOT17-10-SDP\",\n",
    "    \"MOT17-11-SDP\",\n",
    "    \"MOT17-13-SDP\",\n",
    "]\n",
    "test_seqs = [\n",
    "    \"MOT17-01-SDP\",\n",
    "    \"MOT17-03-SDP\",\n",
    "    \"MOT17-06-SDP\",\n",
    "    \"MOT17-07-SDP\",\n",
    "    \"MOT17-08-SDP\",\n",
    "    \"MOT17-12-SDP\",\n",
    "    \"MOT17-14-SDP\",\n",
    "]\n",
    "\n",
    "unzip_url(mot_path, dest=MOT_ROOT_PATH, exist_ok=True)\n",
    "print(f\"Training data saved to {mot_train_path}\")\n",
    "print(f\"Test data saved to {mot_test_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pre-trained, baseline FairMOT model - `all_dla34.pth`- can be downloaded [here](https://drive.google.com/file/d/1udpOPum8fJdoEQm6n0jsIgMMViOMFinu/view). \n",
    "\n",
    "Please upload and save `all_dla34.pth` to the `BASELINE_MODEL` path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below initializes and loads the model using the TrackingLearner class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = TrackingLearner(None, BASELINE_MODEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on Training Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MOT17 provides ground truth annotations for only the training set, so we will be using the training set for evaluation.\n",
    "\n",
    "To evaluate FairMOT on this dataset, we take advantage of the [py-motmetrics](https://github.com/cheind/py-motmetrics) repository. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-02-SDP.txt\n",
      "Evaluate seq: MOT17-02-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-04-SDP.txt\n",
      "Evaluate seq: MOT17-04-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-05-SDP.txt\n",
      "Evaluate seq: MOT17-05-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-09-SDP.txt\n",
      "Evaluate seq: MOT17-09-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-10-SDP.txt\n",
      "Evaluate seq: MOT17-10-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-11-SDP.txt\n",
      "Evaluate seq: MOT17-11-SDP\n",
      "loaded ./models/all_dla34.pth, epoch 10\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-13-SDP.txt\n",
      "Evaluate seq: MOT17-13-SDP\n",
      "              IDF1   IDP   IDR  Rcll  Prcn  GT  MT  PT ML   FP    FN IDs    FM  MOTA  MOTP IDt IDa IDm\n",
      "MOT17-02-SDP 63.9% 77.4% 54.4% 68.5% 97.4%  62  22  31  9  344  5855 183   656 65.7% 0.193  98  33  13\n",
      "MOT17-04-SDP 83.7% 86.1% 81.3% 86.3% 91.4%  83  51  20 12 3868  6531  32   201 78.1% 0.171   5  20   2\n",
      "MOT17-05-SDP 75.9% 82.9% 70.1% 81.2% 96.0% 133  63  59 11  236  1303  79   207 76.6% 0.199  83  26  40\n",
      "MOT17-09-SDP 65.4% 71.6% 60.2% 81.1% 96.5%  26  19   7  0  158  1006  52   105 77.2% 0.165  37  12   7\n",
      "MOT17-10-SDP 65.0% 72.1% 59.2% 78.8% 96.0%  57  32  25  0  418  2721 149   404 74.4% 0.213  89  43  14\n",
      "MOT17-11-SDP 85.6% 87.9% 83.4% 90.4% 95.2%  75  52  19  4  426   910  38   134 85.4% 0.157  24  19  13\n",
      "MOT17-13-SDP 77.0% 82.0% 72.5% 83.9% 94.8% 110  74  29  7  534  1878  86   373 78.5% 0.205  72  26  35\n",
      "OVERALL      76.8% 82.3% 71.9% 82.0% 93.9% 546 313 190 43 5984 20204 619  2080 76.1% 0.182 408 179 124\n"
     ]
    }
   ],
   "source": [
    "strsummary = tracker.eval_mot(\n",
    "    conf_thres=CONF_THRES,\n",
    "    track_buffer=TRACK_BUFFER,    \n",
    "    data_root=mot_train_path,\n",
    "    seqs=train_seqs,\n",
    "    result_root=RESULT_ROOT,\n",
    "    exp_name=EXP_NAME,\n",
    ")\n",
    "print(strsummary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on Test Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For evaluating a model on the testing dataset, the MOT Challenge provides the [MOT evaluation server](https://motchallenge.net/instructions/). Here, a user can upload and submit a txt file of prediction results; the service will return metrics. After uploading our results to the [MOT17 evaluation server](https://motchallenge.net/results/MOT17/?det=Private), we can see a MOTA of 68.5 using the 'all_dla34.pth' baseline model.\n",
    "\n",
    "<img src=\"media/mot_results.PNG\" style=\"width: 737.5px;height: 365px\"/>\n",
    "\n",
    "The reported evaluation results from [FairMOT paper](https://arxiv.org/abs/2004.01888) with test set are as follows:\n",
    "\n",
    "| Dataset | MOTA   | IDF1 | IDS | MT | ML | FPS |\n",
    "|------|------|------|------|------|------|------|\n",
    "|   MOT16  | 68.7| 70.4| 953| 39.5%| 19.0%| 25.9|\n",
    "|   MOT17  | 67.5| 69.8| 2868| 37.7%| 20.8%| 25.9|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-01-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-03-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-06-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-07-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-08-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-12-SDP.txt\n",
      "Saved tracking results to ./results/MOT_val_all_dla34/MOT17-14-SDP.txt\n"
     ]
    }
   ],
   "source": [
    "tracker.eval_mot(\n",
    "    conf_thres=CONF_THRES,\n",
    "    track_buffer=TRACK_BUFFER,    \n",
    "    data_root=mot_test_path,\n",
    "    seqs=test_seqs,\n",
    "    result_root=RESULT_ROOT,\n",
    "    exp_name=EXP_NAME,\n",
    "    run_eval=False,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "356.258px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
